Skip to content

feat(observability): add opt-in Langfuse and Helicone exports - #51

Merged
ALikesToCode merged 2 commits into
integrate/gateway-waves-1-4from
feat/observability-exports
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
integrate/gateway-waves-1-4from
feat/observability-exports

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in exports of content-free request observations to Langfuse and Helicone collectors. OBSERVABILITY_EXPORTERS_JSON defaults to []. While it is unset or empty, nothing is sent and the existing OTLP export is byte-for-byte unchanged.

  • Configuration:
    • Each exporter names a type (langfuse or helicone), an HTTPS endpoint, its exact allowed_origins and a credential_env.
    • Wildcards, URL credentials, queries, fragments and redirects are refused.
    • Any malformed entry disables every exporter and logs one warning, without the setting's value.
  • Credentials:
    • credential_env must name a variable ending in _EXPORT_CREDENTIAL, so a provider or admin key can't be sent to a collector by mistake.
    • Langfuse uses public:secret as Basic auth. Helicone uses a Bearer token.
    • Credentials appear only in the outbound auth header, never in bodies, status or logs.
  • Observation contents:
    • Included: request and correlation IDs, a hashed principal, the selected model, the kind, status, a fixed error class, timings, nullable tokens and cost, and their provenance.
    • Excluded: prompts, output, tools, key prefixes, raw headers and upstream error text.
    • Unknown usage stays unknown; zero is not the same as unknown.
  • Delivery:
    • In-memory queue of 1,000 observations, batches of 50, a 2 s timeout, and at most two retries (transport errors, 429 and 5xx only).
    • Langfuse 207 receipts acknowledge only the IDs that actually succeeded.
    • Export failures never touch the generation, its accounting or its retry permission.
  • Durability: none. A restart or Worker eviction can lose queued observations, and this is documented. No D1 or R2 storage is added.
  • Deduplication: a request that passes through the Worker into Flask is exported once.

Details are in docs/observability-exports.md.

Coordinator edits in this PR

  • Restricted credential_env to *_EXPORT_CREDENTIAL names in both runtimes. The builder's version accepted any variable name. Tests now include GEMINI_API_KEY as a rejected name.
  • Shared registrations:
    • tests/test_observability_exports_worker.mjs in test:worker;
    • worker/observability-export.mjs in the edge module inventory;
    • OBSERVABILITY_EXPORTERS_JSON, LANGFUSE_EXPORT_CREDENTIAL and HELICONE_EXPORT_CREDENTIAL in the Container env keys.

Left for the Wave 5 integration

The native Worker hook needs the execution context for waitUntil. The integration will:

  • register createNativeObservabilityHook(env, ctx) in nativeGenerationFetch;
  • pass ctx to recordNativeUsage.

Until then, native Worker observations are buffered but not flushed. Flask exports already run through TelemetryExporter.submit.

Tests

  • tests/test_observability_exports.py and tests/test_observability_exports_worker.mjs cover:
    • strict configuration and destination rules;
    • content-free schemas;
    • null preservation;
    • retry limits;
    • 207 receipts;
    • credential absence;
    • queue bounds;
    • deduplication;
    • unchanged OTLP output.
  • Full Python suite: 3,933 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,471 passed. npm run test:integration: 7 passed.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • Live collectors were not called. The tests use fake transports.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. It is based on integrate/gateway-waves-1-4, the Wave 4 integration branch. The README line comes with the Wave 5 integration PR.

Send content-free request observations to explicitly configured Langfuse and Helicone collectors when OBSERVABILITY_EXPORTERS_JSON lists them. Delivery uses a bounded in-memory queue with retries limited to transport failures, 429 and 5xx, and counts acknowledgements per exporter. Credentials come from variables ending in _EXPORT_CREDENTIAL. With the setting unset, nothing is sent and OTLP export is unchanged.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy f96656f Oct 09 2026, 09:26 AM

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b1c6c709-73da-4a11-bfd4-0d881ba577eb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…o feat/observability-exports

# Conflicts:
#	package.json
@ALikesToCode
ALikesToCode merged commit fb6ed66 into integrate/gateway-waves-1-4 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/observability-exports branch October 26, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant